Robotic repair or maintenance of an asset

ABSTRACT

A method includes receiving, via at least one sensor of a robot, sensor data indicating one or more characteristics of an asset. The method includes detecting, based on the sensor data, an existing or imminent defect of the asset. The method includes fabricating a part suitable for use in correcting the defect. The structure of the part is derived using one or both of a digital representation of the asset generated using the sensor data or stored reference data related to the asset.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/343,615, entitled “ROBOT SYSTEM FOR ASSET HEALTH MANAGEMENT”,filed May 31, 2016, and U.S. Provisional Patent Application No.62/336,332, entitled “ROBOT SYSTEM FOR ASSET HEALTH MANAGEMENT”, filedMay 13, 2016, which are both herein incorporated by reference in theirentirety for all purposes.

BACKGROUND

The subject matter disclosed herein relates to asset management, andmore particularly, to monitoring and managing health of an asset using arobotic system.

Various entities may own or maintain various types of assets as part oftheir operation. Such assets may include physical or mechanical devicesor structures, which may in some instances, have electrical and/orchemical aspects as well. Such assets may be used or maintained for avariety of purposes and may be characterized as capital infrastructure,inventory, or by other nomenclature depending on the context. Forexample, assets may include distributed assets, such as a pipeline or anelectrical grid as well as individual or discrete assets, such as anairplane, a tower, or a vehicle. Assets may be subject to various typesof defects (e.g., spontaneous mechanical defects, electrical defects aswell as routine wear-and-tear) that may impact their operation. Forexample, over time, the asset may undergo corrosion or cracking due toweather or may exhibit deteriorating performance or efficiency due tothe wear or failure of component parts.

Typically, one or more human inspectors may inspect, maintain, andrepair the asset. For example, the inspector may locate corrosion on theasset and clean the corrosion from the asset. However, depending on thelocation, size, and/or complexity of the asset, having one or more humaninspectors performing inspection of the asset may take away time for theinspectors to perform other tasks. Additionally, some inspection tasksmay be dull, dirty, or may be otherwise unsuitable for a human toperform. For instance, some assets may have locations that may not beaccessible to humans due to height, confined spaces, or the like.Further, inspections may be performed at times that are based onschedules resulting in either over-inspection or under-inspection.Accordingly, improved systems and techniques for managing the health ofvarious types of assets are desirable.

BRIEF DESCRIPTION

Certain embodiments commensurate in scope with the originally claimeddisclosure are summarized below. These embodiments are not intended tolimit the scope of the claimed disclosure, but rather these embodimentsare intended only to provide a brief summary of possible forms of thedisclosure. Indeed, embodiments may encompass a variety of forms thatmay be similar to or different from the embodiments set forth below.

In a first embodiment, a method includes receiving, via at least onesensor of a robot, sensor data indicating one or more characteristics ofan asset, detecting, based on the sensor data, an existing or imminentdefect of the asset, and fabricating a part suitable for use incorrecting the defect, wherein the structure of the part is derivedusing one or both of a digital representation of the asset generatedusing the sensor data or stored reference data related to the asset.

In a second embodiment, a robotic system configured to monitor an assetincludes at least one robot comprising at least one sensor capable ofdetecting one or more characteristics of an asset and at least oneeffector capable of performing a repair or maintenance operation on theasset, and a processing system including at least one processoroperatively coupled to at least one memory, wherein the at least oneprocessor is configured to receive, via at least one sensor of a robot,sensor data indicating one or more characteristics of an asset, detect,based on the sensor data, an existing or imminent defect of the asset,and fabricate a part suitable for correcting the defect using one orboth of a digital representation of the asset or stored reference datarelated to the asset.

In a third embodiment, a non-transitory, computer readable mediumincludes instructions configured to be executed by a processor of arobotic system comprising at least one robot, wherein the instructionsinclude instructions configured to cause the processor to receive, viaat least one sensor of the at least one robot, sensor data indicatingone or more characteristics of an asset, detect, based on the sensordata, an existing or imminent defect of the asset, and perform acorrective action that corrects the defect using one or both of adigital representation of the asset or stored reference data related tothe asset. For example, the instructions may include instructions tofabricate a part suitable for correcting the defect.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure will become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a perspective view of a robotic system with a set of robots tomonitor and manage the health of an asset, in accordance with aspects ofthe present disclosure;

FIG. 2 is a block diagram of the robotic system of FIG. 1 having asecond set of robots to manage another asset with a control system, inaccordance with aspects of the present disclosure; and

FIG. 3 is a flow diagram of a process performed by a controller of thecontrol system of FIG. 2 to manage asset health, in accordance withaspects of the present disclosure;

FIG. 4 is a flow diagram of another process performed by the controllerwhen performing the process of FIG. 3, in accordance with aspects of thepresent disclosure;

FIG. 5 is a flow diagram of another process performed by the controllerwhen performing the process of FIG. 3, in accordance with aspects of thepresent disclosure; and

FIG. 6 is a schematic diagram of a user interface displayed to a user ofthe control system of FIG. 2, in accordance with aspects of the presentdisclosure.

DETAILED DESCRIPTION

The subject matter disclosed herein relates to managing repair and/ormaintenance of an asset with a robotic system. Such an approach may beuseful in monitoring or repairing assets associated with variousentities, including business or corporate entities, governments,individuals, non-profit organizations, and so forth. As discussedherein, such assets may be generally discrete or limited in their extent(e.g., a vehicle such as a plane, helicopter, ship, submersible, spacelaunch vehicle, satellite, locomotive, and so forth) or may begeographically distributed (e.g., a road or rail track, a port orairport, a pipeline or electrical infrastructure, a power generationfacility or manufacturing plant, and so forth). The present approach asdescribed herein may be used to monitor and maintain assets of thesetypes (as well as others not listed) in an autonomous or semi-autonomousmanner using robotic intermediaries. As discussed herein, the roboticintermediaries may be used to facilitate one or both of healthmonitoring of the asset and repair, remediation, or improvement of theasset with limited or no human support.

With this in mind, it will be appreciated that in a variety of fields,assets, such as distributed assets and/or individual assets may be usedto perform any number of operations. Over time, assets may deterioratedue to weather, physical wear, or the like. For example, over months oryears, one or more components of an asset may wear or deteriorate due torain and wind or other environmental conditions or due to inadequatemaintenance. Alternatively, in some instances, spontaneous failures ofone or more components or systems of an asset may occur which may beunrelated to wear or maintenance conditions but may instead beattributable to an undetected defect or an unknown stressor. Regardlessof whether an asset defect is due to gradual process or a suddenoccurrence, the health of the asset depends on identifying andaddressing such defects in a timely and effective manner.

In conventional approaches, one or more human agents (e.g., fieldengineers, operators, or other users of the asset) may inspect the assetfor wear at limited intervals to maintain health of the asset and/or toreplace parts that appear worn. However, the human agents may be unableto inspect components or locations that may not be easily accessible tohumans, such as below the waterline of a marine asset, within a tank orpipe of a pipeline or storage facility, on the exterior surfaces orcomponents of a vehicle in motion (such as a flying plane or helicopter,a moving truck or locomotive), and so forth). Further, when defects arelocated, for some assets human-based repair may require taking the assetout of operation to implement a repair. As such, it is desirable to findimproved ways to monitor and maintain various assets.

With the preceding in mind, in certain embodiments disclosed herein, arobot system may be used to monitor and manage health of an asset thatreduces or eliminates human intervention. A robot may be a machine(e.g., electro-mechanical) capable of carrying out a set of tasks (e.g.,movement of all or part of the machine, operation of one or more type ofsensors to acquire sensed data or measurements, and so forth)automatically (e.g., at least partially without input, oversight, orcontrol by a user), such as a set of tasks programmed by a computer. Forexample, the robot may include one or more sensors to detect one or morecharacteristics of an asset and one or more effectors to perform anoperation based on a plan to assess the asset. The robot system mayinclude a processing system that includes one or more processorsoperatively coupled to memory and storage components. While this may beconceptualized and described below in the context of a singleprocessor-based system to simplify explanation, the overall processingsystem used in implementing an asset management system as discussedherein may be distributed throughout the robotic system and/orimplemented as a centralized control system. With this in mind, theprocessor may be configured to generate a plan to assess the asset fordefects. For example, the processor may determine a plan based on thetasks (e.g., desired inspection coverage of the asset) and/or resources(e.g., robots) available. Based on the generated plan processor mayimplement the plan by sending signal(s) to the robots providinginstructions to perform the tasks defined in the plan. A controller ofeach robot may process any received instructions and in turn sendsignal(s) to one or more effectors controlled by the respective robot tocontrol operation of the robot to perform the assigned tasks.

In certain embodiments, to perform maintenance actions, the processormay generate, maintain, and update a digital representation of the assetbased on one or more characteristics that may be monitored using robotintermediaries and/or derived from known operating specifications. Forexample, the processor may create a digital representation thatincludes, among other aspects, a 3D structural model of the asset (whichmay include separately modeling components of the asset as well as theasset as a whole). Such a structural model may include material data forone or more components, lifespan and/or workload data derived fromspecifications and/or sensor data, and so forth. The digitalrepresentation, in some implementations may also include operational orfunctional models of the asset, such as flow models, pressure models,temperature models, acoustic models, lifting models, and so forth.Further, the digital representation may incorporate or separately modelenvironmental factors relevant to the asset, such as environmentaltemperature, humidity, pressure (such as in the context of a submersibleasset, airborne asset, or space-based asset). As part of maintaining andupdating the digital representation, one or more defects in the asset asa whole or components of the asset may also be modeled based on sensordata communicated to the processing components.

The sensor data used to generate, maintain, and update the digitalrepresentation, including modeling of defects, may be derived fromsensor data collected using one or more of sensors mounted on robotscontrolled by the processing components and/or by sensors integral tothe asset itself which communicate their sensor data to the processingcomponents. As used herein, the robots used to collect sensor data, aswell as effect repairs, may be autonomous and capable of movement andorientation in one—(such as along a track), two—(such as along connectedroads or along a generally planar surface), or three-dimensions (such asthree-dimensional movement within a body of water, air, or space). Thesensors used to collect the sensor data may vary between robots and/ormay be interchangeable so as to allow customization of robots dependingon need. Example of sensors include, but are not limited to, cameras orvisual sensors capable of imaging in one or more of visible, low-light,ultraviolet, and or infrared (i.e., thermal) contexts, thermistors orother temperature sensors, material and electrical sensors, pressuresensors, acoustic sensors, radiation sensors or imagers, probes thatapply non-destructive testing technology, and so forth. With respect toprobes, for example, the robot may contact or interact physically withasset to acquire data.

The digital representation may incorporate or be updated based on acombination of factors detected from one or more sensors on the robot(or integral to the asset itself). For instance, the processor mayreceive visual image data from image sensors (e.g., cameras) on therobots to create or update a 3D model of the asset to localize defectson the 3D model. Based on the sensor data, as incorporated into the 3Dmodel, the processor may detect a defect, such as a crack, a region ofcorrosion, or missing part, of the asset. For example, the processor maydetect a crack on a location of a vehicle based on visual image datathat includes color and/or depth information indicative of the crack.The 3D model may additionally be used as a basis for modeling otherlayers of information related to the asset. Further, the processor maydetermine risk associated with a potential or imminent defect based onthe digital representation. Depending on the risk and a severity of thedefect, the processor, as described above, may send signal(s) to therobots indicating instructions to repair or otherwise address a presentor pending defect.

In some embodiments, to repair or prevent a defect, the processor maycreate a 3D model of a part or component pieces of the part of the assetneeded for the repair. The processor may generate descriptions ofprintable parts or part components (i.e., parts suitable for generationusing additive manufacturing techniques) that may be used by a 3Dprinter (or other additive manufacturing apparatus) to generate the partor part components. Based on the generated instructions or descriptions,the 3D printer may create the 3D printed part. Further, one or morerobots may be used to repair the asset with the 3D printed part(s).While a 3D printed part is described in this example, other repair orremediation approaches may also be employed. For example, in otherembodiments, the processor may send signal(s) indicating instructions toa controller of a robot to control the robot to spray a part of theasset (e.g., with a lubricant or spray paint) or to replace a part ofthe asset from an available inventory of parts. Similarly, in someembodiments, a robot may include a welding apparatus that may beautonomously employed to perform an instructed repair. In someembodiments, the processor may send signal(s) to a display to indicateto an operator to enable the operator to repair the defect.

With the preceding introductory comments in mind, FIG. 1 shows aperspective view of a robotic system 10 that manages health of an asset12 by inspecting and/or repairing the asset 12. The robotic system 10may include a fleet of robots, such as drones (capable of autonomousmovement in one-, two-, or three-dimensions, including movement with orwithout an attached electrical and/or data tether), machines, computingsystems, and so forth. Each of the robots may receive data via sensorsand/or may control operation of one or more effectors of the robot. Inthe illustrated embodiment, the robotic system 10 includes robots, suchas drones, that each have red-green-blue (RGB) sensors, such as cameras,image sensors, photodiodes, or the like, to generate signals indicatingcharacteristics of the asset 12 when the RGB sensor is directed towardthe asset. In the present disclosure, the drones with RGB sensors arereferred to as a first red-green-blue (RGB) drone 14, a second RGB drone16, and a third RGB drone 18. Alternatively, the drones may be referredto more generally as robots. The first RGB drone 14, the second RGBdrone 16, and the third RGB drone 18 may receive signals indicative ofcolors of an exterior of the asset 12. Further, the robotic system 10includes may include a drone having an infrared (IR) camera, referred toas an IR drone 20. While the robotic system 10 of the illustratedembodiment includes drones, any suitable robot that operates at leastpartially autonomous (e.g., without input from an occupant within thevehicle) may be included in the robotic system 10, such as unmannedaerial vehicles, unmanned ground vehicles (e.g., autonomous trucks orlocomotives), unmanned underwater or surface water vehicles, unmannedspace vehicles, crawling robots, or a combination thereof. Further, therobots may operate in one dimension, two dimensions, or threedimensions. While the illustrated embodiment includes four drones, thisis meant to be an example, and any suitable number of any suitablenumber and types of robots (e.g., drones) may be employed. Additionallyand/or alternatively, the robots may include drones that are manuallyguided by an operator. For example, the operator may have a remotecontrol that sends signal(s) to the manually guided drone to control alocation and/or orientation of the drone.

In some embodiments, it may be desirable to have robots that move(autonomously or under direction) to various positions proximate to theasset 12 to acquire sensor data describing one or more characteristicsof the asset from different perspectives with respect to the asset 12.For example, the RGB drones 14, 16, and 18 may move with respect to theasset 12 to receive signal(s) from the RGB sensors indicating thecharacteristics of the asset 12. That is, the drones 14, 16, and 18 mayfly in an at least partially autonomous manner. For instance, the drones14, 16, and 18 may obtain instructions to control a propeller or wingsof the respective drones to adjust the position of the drone withrespect to the asset such that the respective drone 14, 16, and 18 mayacquire additional characteristics of the asset 12 from anotherperspective. In some embodiments, the instructions may be received froma control system or the instructions may be stored on memory of the RGBdrones 14, 16, and 18. For example, the instructions may instruct eachof the RGB drones 14, 16, and 18 to capture images at regular intervalsin a flight path with respect to the asset 12 with or without continuouscommunication and instruction from a separate controller (e.g., acentralized controller). For example, each of the RGB drones 14, 16, and18 may move along a respective path 22, 24, and 26 that orbits the asset12 and/or directs the RGB sensors towards the asset 12. Similarly, theIR drone 20 may move along a path 20 that orbits the asset and/ordirects the IR sensor towards to asset 12 to enable the IR drone 20 tocapture infrared data indicating depth information of the asset 12.

The robotic system 10 may be self-organizing in which tasks areallocated to various members of a multi-robot team based on each of therobots capabilities. For example, a control system may include acontroller that acquires a list of robots with each capability of eachrobot. The controller may determine task assignments of each robot basedon the respective capabilities of each robot. For example, a light dronehaving more flight endurance may be assigned by the controller toperform rough identification of anomalies. Another drone carrying a highresolution camera having less flight time may be assigned by thecontroller to move to specific locations to capture high resolutionimagery. The controller may send signal(s) to each of the dronesindicating instructions to perform the assigned tasks based on thecapabilities of each robot.

Moreover, the robotic system 10 may include inspection systems 28 (e.g.,video cameras) positioned in locations proximate to the asset 12 toacquire various characteristics of the asset 12. Such positionedsystems, unlike the drones described above, may be stationary or havelimited movement from a fixed position, such as being mounted on aremotely controlled moveable arm or having pan, tilt, zoomfunctionality. For example, the inspection systems may acquire colorand/or depth information related to the asset 12 as well as acquireinformation related to the process performed by the drones 14, 16, 18,and 20, such as flight path information with respect to the asset 12,flight path information with respect to each other, altitudeinformation, or the like. Similarly, the robotic system 10 may include amanually controlled drone 30 to acquire various characteristics of theasset 12, similar to those described above regarding the autonomous RGBand IR drones 14, 16, 18, and 20.

The robotic system 10 may plan a mission to inspect the asset andanalyze data from the inspection to find one or more defects. Such aplan may be generated based on available inspection and/or repair assets(e.g., what robots are available having what sensing modalities or whichcan be outfitted with what sensing modalities, what robots are availablehaving what repair modalities, and stationary or integral sensor data isavailable for the asset, and so forth) as well as on the age and/orinspection and repair history of the asset. FIG. 2 is a block diagram ofthe robotic system 10 having a second set of robots that each includeone or more sensors and one or more effectors. In the illustratedembodiment, the robotic system 10 includes a control system 34, a firstrobot 36, a second robot 38, a third robot 40, and a three dimensional(3D) printer. Further, in the example shown in FIG. 2, the first robot36 may be an RGB drone, the second robot 38 may be an autonomousvehicle, and the third robot 40 may be a manipulator system. While therobotic system 10 includes an RGB drone, an autonomous vehicle, and amanipulator system, the robots used in FIG. 2 are simply meant to be anexample, and any robots (e.g., crawling robot, underwater robot,manually controlled robot, etc.) suitable may be included. The robots36, 38, and 40 include a first processing system 42, a second processingsystem 44, and a third processing system 46, respectively. While therobotic system 10 may include the centralized control system 34 as shownin FIG. 2, in other embodiments, parts of the planning and/or controlmay be distributed to each of the processing systems of the roboticsystem 10. Further, while three processing systems are shown, it shouldbe appreciated that any suitable number of processing systems may beused.

In the illustrated embodiment, the control system 34, the firstprocessing system 42, the second processing system 44, the thirdprocessing system 46 and the 3D printer 48 each include a controller 50,52, 54, 56, and 58, respectively. Each controller 50, 52, 54, 56, and 58includes a processor 60, 62, 64, 66, and 68, respectively. Thecontrollers 50, 52, 54, 56, and 58 may also include one or more storagedevices and/or other suitable components, such as the memory devices 70,72, 74, 76, and 78, respectively, operatively coupled to the processors60, 62, 64, 66, and 68, respectively, to execute software, such assoftware for controlling the vehicles (e.g., drones, autonomousvehicles, etc.), detecting defects of the asset 12, repairing and/ormaintaining the asset 12, and so forth. Moreover, the processors 60, 62,64, 66, and 68 may each include multiple processors, one or more“general-purpose” microprocessors, one or more special-purposemicroprocessors, and/or one or more application specific integratedcircuits (ASICS), or some combination thereof. For example, eachprocessor 60, 62, 64, 66, and 68 may include one or more reducedinstruction set (RISC) processors.

Each memory device 70, 72, 74, 76, and 78 may include a volatile memory,such as random access memory (RAM), and/or a nonvolatile memory, such asread-only memory (ROM). Each memory device 70, 72, 74, 76, and 78 maystore a variety of information that may be used for various purposes.For example, each memory device 70, 72, 74, 76, and 78 may storeprocessor-executable instructions (e.g., firmware or software) for therespective processors 60, 62, 64, 66, and 68 to execute, such asinstructions for controlling the vehicles (e.g., drones, autonomousvehicles, etc.), detecting defects of the asset 12, repairing and/ormaintaining the asset 12, and so forth. The storage device(s) (e.g.,nonvolatile storage) may include ROM, flash memory, a hard drive, or anyother suitable optical, magnetic, or solid-state storage medium, or acombination thereof. The storage device(s) may store data (e.g., plannedflight paths, sensor data, etc.), the model of the asset used for healthmanagement, instructions (e.g., software or firmware for controlling thevehicle, etc.), and any other suitable data.

Further, the control system 34, the first processing system 42, thesecond processing system 44, the third processing system 46 and the 3Dprinter 48 may each include a radio frequency (RF) antenna 80, 82, 84,86, and 88, respectively, to communicate with each other. Each of thecontrollers 50, 52, 54, 56, and 58 may communicate using any suitablestandard, such as WiFi (e.g., IEEE 802.11), ZigBee (e.g., IEEE802.15.4), or Bluetooth, among others. For example, the first processingsystem 42 of the RGB drone 36 may send signal(s), via the antenna 82, tothe antenna 80 of the control system 34 indicative of a position of theRGB drone 36. The control system 34 may send signal(s), via the antenna80, to the antenna 82 of the first processing system 42 of the RGB drone36 indicative of instructions to control the RGB drone 36 based on theposition of the RGB drone 36. For instance, each of the controllers 50,52, 54, 56, and 58 may communicate with each other to synchronizeinspection of the asset 12. That is, the flight patterns (e.g.,direction, distance, and timing) of drones may be synchronized with oneanother to prevent drones from interfering with one another whileinspecting the asset 12.

Each of the processing systems 42, 44, and 46 may include spatiallocating devices 90, 92, and 94, respectively, which are each mounted tothe respective robot, and configured to determine a position of thedrone 36, the autonomous vehicle 38, and the ground robot 40,respectively. As will be appreciated, the spatial locating devices 90,92, and 94 may include any suitable system configured to determine theposition of the drone 36, the autonomous vehicle 38, and the groundrobot 40, respectively, such as global positioning system (GPS)receivers, for example. In certain embodiments, the processing systems42, 44, and 46 may receive signal(s) via one or more sensors 96, 98, and100, respectively, indicative of visual inputs of the environment. Eachof the respective processors 62, 64, and 68 may generate a map of theenvironment and localize the respective robot 36, 38, or 40 within themap. Further, localization may include an absolute position (e.g., fixedglobal coordinate system or fixed local coordinate system) as well asposition in relation to the asset (e.g., orientation, distance, etc.).

Each of the processing systems 42, 44, and 46 may include one or moresensors 96, 98, and 100 that send signal(s) to the respectivecontrollers 52, 54, and 56 to facilitate control of the respectiverobots 36, 38, and 40 as well as to acquire data indicative of variousproperties of the asset 12. The sensors 96, 98, 100 may include infraredsensors, ultrasonic sensors, magnetic sensors, thermal sensors,radiation detection sensors, imaging sensors (e.g., RGB sensors), LightDetection and Ranging (LIDAR) sensors, or the like. Further, each robot36, 38, and 40 may include one or more types of sensors.

Moreover, each of the robots 36, 38, and 40 may include one or moreeffectors, such as actuators, motors, or other controls. For example,the robot 36 may include one or more motors 102 that control operationof the robot 36. Each of the robots 36, 38, and 40 may be self-powered(e.g., an engine and/or battery) and/or receive power from another powersource (e.g., via a power tether). Further, the controller 52 may sendsignal(s) to the motors 102 of the robot 36 to control a speed of therotor of the motor 102, thereby controlling the position of the robot36. For example, the controller may send signal(s) indicatinginstructions to increase or decrease speed of one or more of the rotorsof the motors 102 to adjust yaw, pitch, roll, or altitude, of the robot36.

The controller 54 of the robot 38 may generate and send signal(s) tocontrol one or more operations of the robot 38. For instance, thecontroller 56 may send signal(s) to a steering control system 104 tocontrol a direction of movement of the robot 38 and/or to a speedcontrol system 106 to control a speed of the robot 38. For example, thesteering control system 104 may include a wheel angle control system 106that rotates to one or more wheels and/or tracks of the robot 38 may becontrolled to steer the robot 38 along a desired route. By way ofexample, the wheel angle control system 104 may rotate frontwheels/tracks, rear wheels/tracks, and/or intermediate wheels/tracks ofthe robot 38, either individually or in groups. In certain embodiments,a differential braking system may independently vary the braking forceon each lateral side of the robot 38 to direct the robot 38 along thedesired route. Similarly, a torque vectoring system may differentiallyapply torque from an engine to wheels and/or tracks on each lateral sideof the robot 38, thereby directing the robot 38 along a desired route.While the illustrated embodiment of the steering control system 104includes the wheel angle control system 106, it should be appreciatedthat alternative embodiments may include one, two, or more of thesesystems, among others, in any suitable combination.

In the illustrated embodiment, the speed control system 106 may includean engine output control system 110 and/or a braking control system 112.The engine output control system 110 is configured to vary the output ofthe engine to control the speed of the robot 38. For example, the engineoutput control system 110 may vary a throttle setting of the engine, afuel/air mixture of the engine, a timing of the engine, other suitableengine parameters to control engine output, or a combination thereof.Furthermore, the braking control system 112 may adjust braking force,thereby controlling the speed of the robot 38. While the illustratedautomated speed control system 106 includes the engine output controlsystem 110 and the braking control system 129, it should be appreciatedthat alternative embodiments may include one of these systems, amongother systems.

The robot 40 may include a mechanism to repair, replace, or otherwisemaintain the asset 12, such as a manipulator, magnet, suction system,sprayer, or lubricator. In the illustrated embodiment, the robot 40includes a manipulator arm 114, such as electronic, hydraulic, ormechanical arm. Further, robot 40 includes an effector 116, such as aclamp, a container, a handler, or the like. The manipulator arm 114 andthe effector 116 may operate in conjunction with each other to maintainthe asset 12. As an example, the controller 56 of the robot 40 may sendsignal(s) indicating instructions to control one or more motors 118 ofthe manipulator arm 114 and the effector 116 to control the position ofthe manipulator arm 114 and the effector 116 to perform the desiredoperation. As will be appreciated, the controller 56 may send signal(s)indicating instructions to cause the motors to move the manipulator arm114 to a position and/or to secure a 3D printed part 119 onto a defect121 of the asset 12.

In some embodiments, controller 50 may determine a plan that instructsthe robots 36, 38, 46, and 48 to address the defect (e.g., repair,remediate, or otherwise prevent). The plan may include one or more tasksto be performed by the robots 36, 38, 46, and 48. One or more of thecontrollers 50, 52, 54, 56, and 58 may determine a path (e.g., distance,direction, and/or orientation) along which one or more of the robots 36,38, 46, and 48 is moved to address the defect. For example, thecontroller 50 may send signal(s) to the robots 36, 38, 46, and 48indicative of one or more tasks to spray a part of the asset, weld apart of the asset, replace a part of the asset 12 from an inventory ofparts or with a 3D printed part, or the like. Upon addressing the defect(e.g., applying a patch, replacing a part, or spraying a part), thecontroller 50 may determine a plan that instructs the robots 36, 38, 46,and 48 to acquire data to confirm the sufficiency of the repair orpreventative maintenance, e.g., indicative that the defect wasaddressed. For instance, the controller 50 may send signal(s) encodingor conveying instructions to control the robots 36, 38, 46, and 48 totravel along a path planned with respect to the asset 12. The controller50 may acquire sensor data from the sensors 96, 98, and 100 indicativeof one or more characteristics of the asset 12 (e.g., via thetransceivers 80, 82, 84, 88). The controller 50 may then adjust the planto monitor the addressed defect of the asset 12 by adjusting or addingone or more tasks to the plan to acquire additional data related to theasset 12. Upon acquiring data indicative of the defect being addressed,the controller 50 may then send signals to a display 130 to display datarelated to the asset 12, such as detected defects, potential defects,recommendations, repairs, replacement parts, or the like, to anoperator. Further, in some embodiments, the robots may be monitoredand/or controlled by an operator from the control system 34 via the userinterface 128 (e.g., touchscreen display).

The robotic system 10 may include a 3D printer 48 that prints a 3Dprinted part 119. While a 3D printer is described in detail, this ismeant to be an example. In certain embodiments, the 3D model may be sentto another suitable fabrication device capable of fabricating the partusing additive manufacturing in which a device deposits particles to theasset or another location. For example, the particles may be depositedto a location in successive layers to create an object. The 3D printer48 may include a gantry 120 or other structure that supports a printerhead having an extruder 122 that moves across a build platform. The 3Dprinter 48 may also include one or more motors 124 (e.g., steppermotors) that move the extruder 122 with respect to the build platform.For example, the processor 66 may send signal(s) indicating instructionsto control the one or more motors 124 and the extruder 122 to heat asource material 126 and extrude successive layers of the source material126 to create the 3D printed part 119. As will be appreciated, there area various types of 3D printers 48 that may print 3D printed parts in anysuitable manner.

The control system 34 may include a user interface 128 having a display130 to display data related to the asset 12, such as detected defects,potential defects, recommendations, repairs, replacement parts, or thelike, to an operator. Further, in some embodiments, the robots may bemonitored and/or controlled by an operator from the control system 34via the user interface 128 (e.g., touchscreen display).

Each of the robots 36, 38, and 46 may include a collision avoidancesystem 148, 150, and 152, respectively. The collision avoidance systems148, 150, and 152 may include circuitry and/or instructions (e.g.,processor-executable code) to control the sensors 96, 98, and 100 andmotors 102 and 118 of the robots 36, 38, and 46. For example, if anobstacle is detected while the robot 36 is traveling along a pathprovided by the plan, the collision avoidance system 148 on-board therobot 36 may send signals to instruct the motors 102 to control therobot 36 based on a location of the obstacle. That is, the controller 52may determine, via the collision avoidance system 148, a path for therobot 36 to travel that avoids interacting with the obstacle while stillcompleting the tasks assigned to the robot 36.

As mentioned above, the robotic systems 10 of FIGS. 1 and 2 are meant tobe examples, and any suitable combination of robots, including as few asone robot, may be used. Further, as described in detail below, theprocessor 60 of the control system and/or the processor 62 of the robot36 are used as examples, and any suitable combination of robots and/orcontrol systems may be used. For example, some of the steps performed bythe control system may be distributed and performed by the processors62, 64, and 68 of the respective robots 36, 38, and 40. In someembodiments, the processor 60 may determine inspection, maintenance, orrepair actions to be performed by the robots 36, 38, and 40 based on thedigital representation. For instance, the processor 60 may determine atime, schedule, or location at which to perform the inspection,maintenance, or repair, based on the digital representation. Further,the processor 60 may predict health of the asset by comparing thedigital representation to data of other assets. That is, the processor60 may use domain knowledge of the digital representation to predictwhen a defect is likely to occur on the asset 12. For instance, theprocessor 60 may perform an inspection based on the digitalrepresentation that indicates a prediction of a condition of the asset.As such, the processor may perform inspection or maintenance at timesbased on the condition of the asset, thereby reducing time spent oninspection or maintenance as compared to inspections or maintenanceactions performed according to a schedule.

FIG. 3 shows a high level flow diagram of a method 134 that the roboticsystem 10 may perform to manage the asset 12 to reduce or eliminatehuman intervention and improve the lifespan of the asset 12. At block136, the robotic system 10 may first obtain an asset 12 for inspection.The manner in which robotic system 10 obtains the asset 12 may depend onthe type of asset. For example, the robots may move to certain assets 12(e.g., an oil pipeline, power transmission lines, etc.) to assess theasset for defects (e.g., cracks in an oil pipeline). At block 138, theprocessor 60 may determine a plan to assess the asset 12 for defects.The plan to assess the asset 12 may be any suitable plan. For example,the plan may include one or more tasks based on the resources (e.g.,available robots) and/or the asset 12. At block 140, the robotic systemmay then detect and assess a defect 121 associated with the asset 12. Atblock 142, the robotic system 10 may manage the asset based on thedefect 121. For example, the robotic system may repair and/or replaceone or more parts of the asset 12 based on the severity of the defect12. Each of blocks 140 and 142 are explained below.

FIG. 4 shows a process performed at block 140 by one or more of theprocessors 60, 62, 64, and 68 to perform automated defect recognition(ADR). Upon performing the plan described above, at block 162, eachcontroller 52, 54, and 56 of the robots 36, 38, and 40 may acquire datarelated to one or more characteristics of an asset 12. As mentionedabove, the data may be acquired via the respective sensors 96, 98, and100 of the robots 36, 38, and 40. In some embodiments, the data mayinclude environmental data from one or more environmental sensors otherthan the sensors 96, 98, and 100. Further, the asset 12 may include oneor more sensors to provide information to the robotic system 10.

At block 166, the processor 60 of the control system 34 may receive thedata from the robots 36, 38, and 40 and generate a digitalrepresentation of the asset 12 based on the one or more characteristics.That is, the data collected may be used to build, update, and maintain adigital representation of the asset 12 as described above. Additionallyand/or alternatively, the processor 60 may generate the digitalrepresentation based in part on physics models and/or domain knowledge.The digital representation may include a mathematical model that hasvariables extrapolated from various parts of the asset 12. For example,the processor 60 may generate a digital representation that includesphysical geometry of the asset 12 (e.g., gathered via the sensors 96,98, and 100), a 3D model of the asset 12, materials of the asset 12,lifespan of the asset 12, observed or measured performance of the asset12, or any combination thereof. In certain embodiments, each of therobots 36, 38, and 40 may, solely or collaboratively, generate a digitalrepresentation of all or part of the asset 12 based on the acquireddata.

At block 168, one or more of the processors 60, 62, 64, and 68 maydetect the defect 121 of the asset 12 based on the one or morecharacteristics. For example, the defect 121 may include a crack in thephysical structure of the asset 12, corrosion on the asset 12, debris onthe asset 12, material aging of the asset 12, missing parts of the asset12, or any other suitable anomaly of the asset 12. The digitalrepresentation may include a location of the defect with respect togeometry of the asset 12. The processor 60 may recognize the defect 121by comparing the one or more characteristics with prior knowledge of theasset 12 or by analysis of the digital representation against knownparameters or patterns. Further, if a potential defect is detected, thecontroller 50 of the control system 34 may send signal(s) to thecontrollers 52, 54, and 56 indicating instructions to adapt the plans toacquire additional data related to the potential defect. Alternativelyand/or additionally, the controller 50 may send signal(s) to the display130 to display data related to the defect 121 to inform an operator.

At block 170, the processor 60 may determine risk associated with thedefect 121 of the asset 12 based on the severity of the defect, thelocation of the defect, the likelihood of poor performance due to thedefect, among others. Further, depending on the risk associated with thedefect, the processor 60 may determine whether or not to perform amaintenance action. For example, if the processor 60 determines that alikelihood of improved performance from repairing the defect 121 of theasset 12 outweighs the cost associated with repairing the defect, thenthe processor 60 may send signal(s) indicating instructions to performthe maintenance action (block 172). For example, the controller 50 maysend signal(s) to the 3D printer indicating instructions to print a 3Dprinted part, as described in detail below. In some embodiments, themaintenance actions may be related to robot fleet management. That is,the processor 60 may send signal(s) indicating instructions to inspectareas based on previously detected anomalies and the risk of theanomalies. For instance, the processor 60 may send signal(s) indicatinginstructions to inspect an area of the asset 12 that is prone tocracking. Similarly, a maintenance action related to robot fleetmanagement may relate to setting or modifying an inspection interval,specifying certain types of robots and/or sensors be deployed for aninspection, acquiring operation or functional data related to assetperformance that might relate to a possible or pending defect, and soforth.

FIG. 5 shows an example of a process performed at block 142 by one ormore of the processors 60, 62, 64, and 68 to manage the asset 12 basedon the defect 121. The example shown describes a process of 3D printinga repair part. The process described is meant to be an example, andother processes may be performed to manage the asset 12, such asreplacing, removing, cleaning, welding, or lubricating a part of theasset 12, among others. As another example, the processor 60 may sendsignal(s) to the display 130 indicating instructions to display arecommendation to an operator. As mentioned above, the processor 60 maydetermine an action to be performed, such as a maintenance and/or arepair operation. At block 182, the processor may create a 3D model of arepair to a part of an asset 12 from the digital representation of theasset 12. For example, each of the robots 36, 38, and 40 may acquirevisual image data from image sensors as well as depth information frominfrared sensors. The robot controllers 52, 54, and 56 may sendsignal(s), via the antennas 82, 84, and 88, to the controller 50indicating the visual image data and depth information of the asset 12.The controller 50 may receive the signal(s) via the antenna 80 and theprocessor 60 may construct a 3D model of the repair to the part of theasset 12 based on the visual image data and depth information. Forinstance, the 3D model may constructed by having a part library thatincludes each parts of the asset 12. Further, the 3D model associatedwith the part having the defect may be printed to replace the existingpart of the asset 12. As another example, the asset 12 may have knownrepair parts that are associated with defects from prior inspections.Upon recognizing a defect that shares characteristics of the priordefect, the processor 60 may select the 3D model from the known repairparts. In some embodiments, the processor 60 augment the 3D model viacoloring based on the sensor data and provide the augmented 3D model toan operator via the display.

In certain embodiments, the processor 60 may create the 3D model basedon domain knowledge regarding the asset 12. For example, the processor60 may be assessing an oil and gas pipeline for defects. Upon locatingan aperture in the oil and gas pipeline, the processor 60 may create a3D model that secures the oil and gas within the pipeline by detectinglocations and distances of edges of the aperture on the pipeline tocreate a 3D model having a size and shape that matches the detectedlocations and distances. Further, the processor 60 may determine whetherthe oil and gas pipeline is liquid tight such that liquids would notleak from the application of the 3D model.

At block 184, the processor 60 may split the 3D model into one or more3D printable parts to meet desired print times and/or based on thesource materials 126 used to print the 3D printed part 119. Theprocessor 60 may send signal(s) to the controller 58 of the 3D printer48 indicating the 3D model to be printed (block 186). At block 188, thecontroller 58 of the 3D printer 48 may receive the 3D model via theantenna 86 and send signal(s) to the motors 124 to control the gantry120 and/or the extruder 122 to create the 3D printed part 119 from the3D model.

At block 190, the robotic system 10 may repair the asset with the 3Dprinted part. For example, in FIG. 2, the controller 58 of the 3Dprinter may send signal(s) to the controller 56 of the robot 40indicating that the 3D printed part 119 is created. Upon creating of the3D printed part, the robot 40 may send signal(s) indicating instructionsto manipulate the manipulator arm 114 and the effector 116 to receivethe 3D printed part 119 and to install the 3D printed part 119 onto thedefect 121 of the asset 12.

Data acquired via the controllers to the asset 12 may be displayed tothe user in a variety of ways. FIG. 6 shows an example of a userinterface 128 displayed on the display 130 to a user of the controlsystem 34 of FIG. 2, in accordance with aspects of the presentdisclosure. The user interface 128 may include a display panel 196 thatdisplays sensor data, such as images, from the robotic system 10 (e.g.,from the controllers 50, 52, 54, 56, and 58 of the robots 34, 36, 38,46, and 48). The user interface 128 may include one or more overlays 198that may overlay features of the data on the display panel 196. Thecontroller 50 may receive signals (e.g., via a touchscreen, a keyboard,a mouse, etc.) indicating a selection of one or more overlays 198. Thecontroller 50 may then send signal(s) indicating instructions to displaya heat map overlaid on a model 200 of the asset 12 in the display panel196 having heat signatures 202 from an IR sensor in an identifying colorto enable the user to recognize the heat signatures 202 on the asset 12.Further, the controller 50 may receive a selection indicative ofinstructions to overlay recognized defects on the display panel 196 orany other overlay suitable for an operator to assess the asset 12, suchcorrosion, cracks, or the like.

Technical effects of the disclosure include management of health of anasset. A robotic system may plan one or more paths for robots to performtasks to acquire characteristics of the asset. The robots may inspectthe asset and receive data from sensors indicating the characteristicsof the asset. A processing system of the robotic system may then detecta defect of the asset. The robotic system may then repair the asset byreplacing a part, 3D printing a part, or performing another maintenanceoperation. For example, the processing system may create a 3D model toprint to repair the asset. The processing system may send signal(s) to a3D printer to print the 3D model. Further, the processing system maydisplay the model of the asset on a display. In certain embodiments, thedisplay may display one or more overlays onto the model to enable anoperator to assess various characteristics of the asset, such as heatsignatures.

This written description uses examples to disclose various embodiments,including the best mode, and also to enable any person skilled in theart to practice the disclosure, including making and using any devicesor systems and performing any incorporated methods. The patentable scopeof the disclosure is defined by the claims, and may include otherexamples that occur to those skilled in the art. Such other examples areintended to be within the scope of the claims if they have structuralelements that do not differ from the literal language of the claims, orif they include equivalent structural elements with insubstantialdifferences from the literal languages of the claims.

The invention claimed is:
 1. A method, comprising: receiving, via atleast one sensor of a robot, sensor data indicating one or morecharacteristics of an asset; detecting, based on the sensor data, adefect associated with a part of the asset; creating or updating a threedimensional (3D) model of the part, wherein a structure of the part isderived using one or both of a digital representation of the assetgenerated using the sensor data or stored reference data related to theasset; and performing a corrective action that corrects the defect. 2.The method of claim 1, wherein the 3D model comprises a location of thedefect with respect to geometry of the 3D model.
 3. The method of claim2, comprising sending the 3D model to a fabrication device configured tofabricate a replacement part based on the 3D model using additivemanufacturing.
 4. The method of claim 3, comprising instructing therobot or another robot to autonomously repair the defect by replacingthe part with the replacement part.
 5. The method of claim 4, wherein alocation of the replacement part on the asset is determined using thesensor data.
 6. The method of claim 1, comprising dividing the 3D modelof the part into a plurality of 3D printable subcomponents based onprint times of each of the 3D printable subcomponents, source materialsused to print each of the 3D printable subcomponents, or any combinationthereof.
 7. A repair system configured to monitor an asset, comprising:at least one robot comprising at least one sensor configured to detectone or more characteristics of an asset and at least one effectorconfigured to perform a repair or maintenance operation on the asset;and a processing system comprising at least one processor operativelycoupled to at least one memory, wherein the at least one processor isconfigured to: receive, via the at least one sensor of the at least onerobot, sensor data indicating the one or more characteristics of theasset; detect, based on the sensor data, a defect of the asset; display,via a display device, a three dimensional (3D) model of the asset,wherein the processor is configured to overlay features of the defectonto the 3D model of the asset; and perform a corrective action thatcorrects the defect using one or both of the 3D model of the asset orstored reference data related to the asset.
 8. The repair system ofclaim 7, wherein the corrective action comprises fabricating areplacement part suitable for correcting the defect.
 9. The repairsystem of claim 7, wherein the 3D model is colored based on the sensordata.
 10. The repair system of claim 7, wherein the processing system isconfigured to determine whether the 3D model is solid.
 11. The repairsystem of claim 10, wherein the processing system is configured to: senda first signal indicative of instructions to display the 3D model on thedisplay device of the repair system; receive a second signal indicativeof modifications to the 3D model to enable the 3D model to become solid;and send a third signal indicative of instructions to print areplacement part from the modified 3D model on a fabrication device. 12.The repair system of claim 7, comprising a fabrication device proximateto a location of the asset or on the asset.
 13. A non-transitory,computer readable medium comprising instructions configured to beexecuted by a processor of a repair system comprising at least onerobot, wherein the instructions comprise instructions configured tocause the processor to: receive, via at least one sensor of the at leastone robot, sensor data indicating one or more characteristics of anasset; detect, based on the sensor data, a defect associated with a partof the asset; generate a three dimensional (3D) model of the asset usingone or both of a digital representation of the asset or stored referencedata related to the asset, wherein the 3D model includes a model of thepart suitable for correction of the defect; divide the model of the partinto a plurality of subcomponents based on an amount of time tofabricate each of the subcomponents, source materials used to fabricateeach of the subcomponents, or any combination thereof; and fabricateeach of the subcomponents for correction of the defect.
 14. Thenon-transitory computer readable medium of claim 13, comprisinginstructions configured to cause the processor to build, update, ormaintain the digital representation of the asset based on the one ormore characteristics, wherein the digital representation of the assetcomprises a mathematical model of parts of the asset.
 15. Thenon-transitory computer readable medium of claim 14, wherein the digitalrepresentation comprises a physical geometry of the asset.
 16. Thenon-transitory computer readable medium of claim 13, wherein the 3Dmodel is displayed on a display and is colored based on the sensor data.17. The non-transitory computer readable medium of claim 16, comprisinginstructions configured to cause the processor to: send a first signalindicative of instructions to display the 3D model on the display of therepair system; receive a second signal indicative of modifications tothe 3D model to enable the 3D model to become solid; and send a thirdsignal indicative of instructions to print a replacement part from themodified 3D model on a fabrication device.
 18. The non-transitorycomputer readable medium of claim 13, wherein the plurality ofsubcomponents collectively form a replacement part, and wherein thenon-transitory, computer readable medium comprises instructionsconfigured to cause the processor to correct the defect by instructingthe at least one robot to replace the part with the replacement part.19. The repair system of claim 7, wherein the features comprisecorrosion on the asset, cracks on the asset, or both.
 20. The repairsystem of claim 8, wherein the corrective action further comprisesreplacing the part on the asset with the replacement part via the atleast one robot.